CFM03226 / P203-0366 

WHAT IS CLAIMED IS: 

1 . An image display apparatus which comprises a 
display device that displays an image to be presented 
to a passenger on a vehicle on a front windshield 

5 portion of the vehicle, comprising: 

vehicle measurement unit adapted to measure a 
position and azimuth of the vehicle on a world 
coordinate system ; 

head measurement unit adapted to measure a 
10 position of a head of the passenger on the vehicle on a 
coordinate system defined in the vehicle; and 

control unit adapted to control the display 
device to display the image at a position according to 
the position and azimuth of the vehicle measured by 
15 said vehicle measurement unit, and the position of the 
head measured by said head measurement unit, on the 
front windshield portion, 

2 . The apparatus according to claim 1 , wherein said 
20 vehicle measurement unit comprises external world image 

sensing unit adapted to sense an image of an external 
world of the vehicle. 

3. The apparatus according to claim 2, wherein said 
25 vehicle measurement unit calculates positions of 

indices in the image, which are included in the image 
of the external world sensed by said external world 
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image sensing unit, and measures the position and 

— cne renxcie oii cue woria coordinate system 
on the basis of the calculated position. 

4. The apparatus according to claim 2, wherein said 
vehicle measurement unit further comprises rotational 
velocity measurement unit adapted to measure rotational 
velocities of right and left rear wheels of the vehicle, 
and 

said vehicle measurement unit calculates 
positions of indices in the image, which are included 
in the image of the external world sensed by said 
external world image sensing unit, and measures the 
position and azimuth of the vehicle on the world 
coordinate system on the basis of the calculated 
positions, and the rotational velocities measured by 
said rotational velocity measurement unit. 



5. The apparatus according to claim 1, wherein said 
20 head measurement unit comprises head image sensing unit 
adapted to sense an image of the head of the passenger 
on the vehicle, and 

said head measurement unit calculates positions 
of indices in the image, which are included in the 
25 image of the head of the passenger sensed by said head 
image sensing unit, and measures the position of the 
head of the passenger on the vehicle on the coordinate 
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system defined in the vehicle on the basis of the 
calculated positions. 

6 . The apparatus according to claim 1 , wherein the 
5 image is an image indicating navigation information. 

7. The apparatus according to claim 1, wherein said 
vehicle measurement unit measures the position and 
azimuth of the vehicle using a nonholonomic constraint. 

10 

8 . A measurement apparatus which is mounted on a 
vehicle and is used to measure a position and azimuth 
of the vehicle on a world coordinate system, 
comprising: 

15 external world image sensing unit adapted to 

sense an image of an external world of the vehicle; 

position calculation unit adapted to calculate 
positions of indices in the image, which are included 
in the image of the external world sensed by said 

20 external world image sensing unit; and 

position/azimuth measurement unit adapted to 
measure the position and azimuth of the vehicle on the 
world coordinate system on the basis of the positions 
of the indices in the image calculated by said position 

25 calculation unit. 
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9 . The apparatus according to claim 8 , further 
comprising rotational velocity measurement unit adapted 
to measure rotational velocities of right and left rear 
wheels of the vehicle, and 

5 said position/azimuth measurement unit measures 

the position and azimuth of the vehicle on the world 
coordinate system further using the rotational 
velocities measured by said rotational velocity 
measurement unit . 

10 

10. A measurement apparatus which measures a position 
and orientation of an image sensing device or an object 
sensed by the image sensing device by detecting indices 
on an image of the object sensed by the image sensing 

15 device, comprising: 

an index selection unit adapted to select indices 
which are to be observed on the sensed image without 
being occluded, 

wherein the position and orientation of the image 
20 sensing device or the object sensed by the image 

sensing device are measured on the basis of information 
of the selected indices. 



11. The apparatus according to claim 10, wherein said 
25 index selection unit comprises: 

index information holding unit adapted to hold 
information associated with each index; and 
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occlusion determination unit adapted to determine 
a presence/absence of occlusion of each index on the 
basis of the information associated with each index 
held by said index information holding unit, and 
5 said index selection unit selects indices which 

are determined to be free from occlusion. 

12. The apparatus according to claim 11, wherein said 
index information holding unit holds at least 
0 position/orientation information of the image sensing 
device or the object upon sensing a template used to 
detect each index, and 

said occlusion determination unit determines the 
presence/absence of occlusion by comparing the 
> position/orientation information of the image sensing 
device or the object held by said index information 
holding unit, and estimated values of a position and 
orientation of the image sensing device or the object. 

13. The apparatus according to claim 11. wherein said 
index information holding unit holds at least 
information associated with a direction of each index, 
and 

said occlusion determination unit determines the 
presence/absence of occlusion by comparing the 
information associated with a direction of each index 
held by said index information holding unit with an 
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estimated value of a direction of the image sensing 
device . 

14. The apparatus according to claim 11, wherein said 
index information holding unit holds at least a 
position of each index, and 

said occlusion determination unit determines the 
presence/absence of occlusion on the basis of the 
position of each index, estimated values of a position 
and orientation of the image sensing device, and shape 
information of a real space. 

15. An image display method executed by an image 
display apparatus which comprises a display device that 
displays an image to be presented to a passenger on a 
vehicle on a front windshield portion of the vehicle, 
comprising: 

a vehicle measurement step of measuring a 
position and azimuth of the vehicle on a world 
coordinate system; 

a head measurement step of measuring a position 
of a head of the passenger on the vehicle on a 
coordinate system defined in the vehicle; and 

a control step of controlling the display device 
to display the image at a position according to the 
position and azimuth of the vehicle measured in the 
vehicle measurement step, and the position of the head 
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measured in the head measurement step, on the front 
windshield portion. 

16. A measurement method which is mounted on a 

5 vehicle and is used to measure a position and azimuth 
of the vehicle on a world coordinate system, 
comprising: 

an external world image sensing step of sensing 
an image of an external world of the vehicle using an 

10 image sensing device; 

a position calculation step of calculating 
positions of indices in the image, which are included 
in the image of the external world sensed in the 
external world image sensing step; and 

15 a position/azimuth measurement step of measuring 

the position and azimuth of the vehicle on the world 
coordinate system on the basis of the positions of the 
indices in the image calculated in the position 
calculation step, 

20 

17, A measurement method which measures a position 
and orientation of an image sensing device or an object 
sensed by the image sensing device by detecting indices 
on an image of the object sensed by the image sensing 

25 device, comprising: 
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an index selection step of selecting indices 
which are to be observed on the sensed image without 
being occluded, 

wherein the position and orientation of the image 
5 sensing device or the object sensed by the image 

sensing device are measured on the basis of information 
of the selected indices . 

18. The method according to claim 17, wherein the 
10 index selection step comprises: 

an index information holding step of holding 
information associated with each index in a memory; and 

an occlusion determination step of determining a 
presence/absence of occlusion of each index on the 
15 basis of the information associated with each index 
held by the memory, and 

the index selection step includes a step of 
selecting indices which are determined to be free from 
occlusion, 

20 

19. The method according to claim 18, wherein the 
memory holds at least information associated with a 
direction of each index, and 

the occlusion determination step includes a step 
25 of determining the presence/absence of occlusion by 

comparing the information associated with a direction 
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of each index held by the memory with an estimated 
value of a direction of the image sensing device, 

20. A program for making a computer function as an 
5 image display apparatus of claim 1 . 

21. A program for making a computer function as a 
measurement apparatus of claim 8 . 

10 22. A program for making a computer execute an image 
display method of claim 15. 

23. A program for making a computer execute a 
measurement method of claim 16 . 

15 

24. A computer readable storage medium storing a 
program of claim 20. 

25. An information processing method which comprises 
20 a detection step of detecting a position of an index in 

a sensed image sensed by image sensing unit adapted to 
sense an image of a real space where the index is laid 
out, and a first calculation step of calculating a 
coordinate position of the index upon projecting the 
25 index in the real space onto the sensed image on the 
basis of the position of the index in the real space 
and a position and orientation of the image sensing 
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unit when at least one of the position of the index in 
the real space and the position and orientation of the 
image sensing unit is obtained based on a measurement, 
and executes a process for determining correspondence 
5 between indices at coordinate positions with a smaller 
distance on the basis of coordinate positions of 
indices detected in the detection step and the 
coordinate position calculated in the first calculation 
step on the sensed image, comprising: 

10 a second calculation step of calculating a value 

using a normal vector to an index of interest, and a 
visual axis vector of the image sensing unit; and 

a determination step of determining, on the basis 
of a range of the value calculated in the second 

15 calculation step, whether or not the process for 

calculating the coordinate position of the index of 
interest on the sensed image in the first calculation 
step is to be executed, 

wherein when it is determined in the 

20 determination step that the process for calculating the 
coordinate position of the index of interest on the 
sensed image in the first calculation step is to be 
executed, the process for determining correspondence 
between indices at coordinate positions with a smaller 

25 distance is executed based on the coordinate position 
of the index of interest calculated in the first 
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calculation step and the coordinate positions of the 
indices detected in the detection step. 

26. The method according to claim 25, wherein the 
5 second calculation step includes a step of calculating 
an angle the normal vector of the index of interest 
makes with the visual axis vector of the image sensing 
unit or a value based on the angle* 

10 27. The method according to claim 26, wherein the 

determination step includes a step of determining that 
the process for calculating the coordinate position of 
the index of interest on the sensed image in the first 
calculation step is to be executed when the angle is 

15 90° + a (a ^ 0) . 

28. The method according to claim 25, further 
comprising: 

a correction step of correcting the position and 
20 orientation of the image sensing unit using a distance 
between the coordinate position calculated in the first 
calculation step and the coordinate position detected 
in the detection step of the determined corresponding 
indices ; 

25 a virtual space image generation step of 

generating an image of a virtual space using the 
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position and orientation corrected in the correction 
step; and 

a composition step of compositing and outputting 
the image of the virtual space generated in the virtual 
5 space image generation step and the sensed image sensed 
by the image sensing unit. 

29. An information processing apparatus which 
comprises image sensing unit adapted to sense an image 

10 of a real space where an index is laid out, detection 
unit adapted to detect a position of the index in a 
sensed image sensed by the image sensing unit, and 
first calculation unit adapted to, when at least one of 
a position of an index in the real space and a position 

15 and orientation of the image sensing unit is obtained 
based on a measurement, calculating a coordinate 
position of the index upon projecting the index in the 
real space onto the sensed image on the basis of the 
position of the index in the real space and the 

20 position and orientation of the image sensing unit, and 
executes a process for determining correspondence 
between indices at coordinate positions with a smaller 
distance on the basis of coordinate positions of 
indices detected by the detection unit and the 

25 coordinate position calculated by the first calculation 
unit, on the sensed image, comprising: 
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second calculation unit adapted to calculate a 
value using a normal vector to an index of interest, 
and a visual axis vector of the image sensing unit; and 

determination unit adapted to determine, on the 
5 basis of a range of the value calculated by said second 
calculation unit, whether or not the process for 
calculating the coordinate position of the index of 
interest on the sensed image by the first calculation 
unit is to be executed, 
10 wherein when said determination unit determines 

that the process for calculating the coordinate 
position of the index of interest on the sensed image 
by the first calculation unit is to be executed, the 
process for determining correspondence between indices 
15 at coordinate positions with a smaller distance is 

executed based on the coordinate position of the index 
of interest calculated by the first calculation unit 
and the coordinate positions of the indices detected by 
the detection unit. 

20 

30. A program for making a computer execute an 
information processing method of claim 25. 

31. A computer readable storage medium storing a 
25 program of claim 30. 
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32. An identification method of an index used to 
measure a position and orientation of an image sensing 
device for sensing an image of a real space, 
comprising: 

5 a position/orientation measuring step of 

measuring roughly a position and orientation of the 
image sensing device; 

an image sensing step of sensing an image of a 
real space including an index using the image sensing 
10 device; 

a first image coordinate calculation step of 
calculating coordinate of the index , which is included 
in the image of the real space obtained in the image 
sensing step, in the obtained image; and 

15 a second image coordinate calculation step of 

calculating a position of the index in a sensed image 
sensed by the image sensing device, whose position and 
orientation are obtained in the position/orientation 
measuring step, 

20 wherein the index is identified on the basis of 

the coordinate of the index calculated in the first 
image coordinate calculation step, the coordinate of 
the index calculated in the second image coordinate 
calculation step, and relationship between a visual 

25 axis vector of the image sensing device obtained in the 
position/orientation measuring step and a normal vector 
of the index. 
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33. An identification method of an index used to 
measure a position and orientation of an object in a 
real space, comprising: 
5 a position/orientation measuring step of 

measuring roughly a position and orientation of the 
object; 

an image sensing step of sensing an image of the 
object using an image sensing device, which is fixed in 

10 position; 

a first image coordinate calculation step of 
calculating coordinate of an index, which is included 
in the image of the object obtained in the image 
sensing step, in the obtained image; and 

15 a second image coordinate calculation step of 

calculating a position of the index in a sensed image 
sensed by the image sensing device, on the basis of 
position and orientation are obtained in the 
position/orientation measuring step , 

20 wherein the index is identified on the basis of 

the coordinate of the index calculated in the first 
image coordinate calculation step, the coordinate of 
the index calculated in the second image coordinate 
calculation step, and relationship between a visual 

25 axis vector of the image sensing device obtained in the 
position/orientation measuring step and a normal vector 
of the index. 
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34. An identification method of an index used to 
measure a position and orientation of an object in a 
real space, comprising: 
5 a first position/orientation measuring step of 

measuring roughly a position and orientation of the 
object ; 

a second position/orientation measuring step of 
measuring roughly a position and orientation of an 
10 image sensing device which senses the object; 

an image sensing step of sensing an image of the 
object using the image sensing device; 

a first image coordinate calculation step of 
calculating coordinate of an index, which is included 
15 in the image of the object obtained in the image 
sensing step, in the obtained image; and 

a second image coordinate calculation step of 
calculating a position of the index in a sensed image 
sensed by the image sensing device, on the basis of a 
20 position and orientation of the object with reference 
to the image sensing device, or of the image sensing 
device with reference to the object, which is obtained 
in the first and second position/orientation measuring 
step, 

25 wherein the index is identified on the basis of 

the coordinate of the index calculated in the first 
image coordinate calculation step, the coordinate of 
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the index calculated in the second image coordinate 
calculation step , and relationship between a visual 
axis vector of the image sensing device obtained in the 
second position/orientation measuring step, and a 
5 normal vector of the index obtained in the first 
position/orientation measuring step . 
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